{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![AIcrowd-Logo](https://raw.githubusercontent.com/AIcrowd/AIcrowd/master/app/assets/images/misc/aicrowd-horizontal.png)\n",
    "\n",
    "# Food Recognition Challenge (Local Evaluation)\n",
    "\n",
    "**Author** : [Sharada Mohanty](mailto:sharada.mohanty@epfl.ch)\n",
    "\n",
    "This notebook walks you through the process of locally evaluating your submissions.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import json\n",
    "import numpy as np\n",
    "import argparse\n",
    "import base64\n",
    "import glob\n",
    "import os\n",
    "from PIL import Image\n",
    "\n",
    "from pycocotools.coco import COCO\n",
    "from cocoeval import COCOeval\n",
    "# Note that, we use a slightly modified version of the official `COCOEval` class, \n",
    "# and it has been included in this repository for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Configuration Variables\n",
    "padding = 50\n",
    "SEGMENTATION_LENGTH = 10\n",
    "MAX_NUMBER_OF_ANNOTATIONS = 10\n",
    "\n",
    "IMAGES_DIR = \"data/val/images\"\n",
    "GROUND_TRUTH_ANNOTATION_PATH = \"data/val/annotations.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin with, we will gather everything we discussed in the [Random Submission](https://github.com/crowdAI/mapping-challenge-starter-kit/blob/master/Random%20Submission.ipynb) notebook, and make it a single function that we can re-use in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_and_save_random_prediction(IMAGE_IDS):\n",
    "    def bounding_box_from_points(points):\n",
    "        \"\"\"\n",
    "            This function only supports the `poly` format.\n",
    "        \"\"\"\n",
    "        points = np.array(points).flatten()\n",
    "        even_locations = np.arange(points.shape[0]/2) * 2\n",
    "        odd_locations = even_locations + 1\n",
    "        X = np.take(points, even_locations.tolist())\n",
    "        Y = np.take(points, odd_locations.tolist())\n",
    "        bbox = [X.min(), Y.min(), X.max()-X.min(), Y.max()-Y.min()]\n",
    "        bbox = [int(b) for b in bbox]\n",
    "        return bbox\n",
    "\n",
    "    def single_segmentation(image_width, image_height, number_of_points=10):\n",
    "        points = []\n",
    "        for k in range(number_of_points):\n",
    "            # Choose a random x-coordinate\n",
    "            random_x = int(random.randint(0, image_width))\n",
    "            # Choose a random y-coordinate\n",
    "            random_y = int(random.randint(0, image_height))\n",
    "            #Flatly append them to the list of points\n",
    "            points.append(random_x)\n",
    "            points.append(random_y)\n",
    "        return [points]\n",
    "\n",
    "    def single_annotation(image_id, number_of_points=10):\n",
    "        \n",
    "        image_file_name = \"{}.jpg\".format(str(image_id).zfill(6))\n",
    "        image_path = os.path.join(IMAGES_DIR, image_file_name)\n",
    "        im = Image.open(image_path)\n",
    "        width, height = im.size\n",
    "        \n",
    "        _result = {}\n",
    "        _result[\"image_id\"] = image_id\n",
    "        _result[\"category_id\"] = 100 # as 100 is the category_id of Building\n",
    "        _result[\"score\"] = np.random.rand() # a random score between 0 and 1\n",
    "\n",
    "        _result[\"segmentation\"] = single_segmentation(width, height, number_of_points=number_of_points)\n",
    "        _result[\"bbox\"] = bounding_box_from_points(_result[\"segmentation\"])\n",
    "        return _result\n",
    "    \n",
    "    predictions = []\n",
    "    for image_id in IMAGE_IDS:\n",
    "        number_of_annotations = random.randint(0, MAX_NUMBER_OF_ANNOTATIONS)\n",
    "        for _idx in range(number_of_annotations):\n",
    "            _annotation = single_annotation(image_id)\n",
    "            predictions.append(_annotation)\n",
    "    \n",
    "    import json\n",
    "    fp = open(\"predictions.json\", \"w\")\n",
    "    fp.write(json.dumps(predictions))\n",
    "    fp.close()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will focus on the validation set, and first generate a random prediction for the validation set.   \n",
    "\n",
    "We will collect the image_ids this time by reading the actual list of files, and then we use the function we just defined to create a random `predictions.json` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "image_paths = glob.glob(os.path.join(IMAGES_DIR, \"*.jpg\"))\n",
    "IMAGE_IDS = [int(os.path.basename(x).replace(\".jpg\", \"\")) for x in image_paths]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "generate_and_save_random_prediction(IMAGE_IDS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing for evaluation\n",
    "\n",
    "Before we can evaluate a submission, we will need the ground truth annotations, which we can load by : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading annotations into memory...\n",
      "Done (t=0.04s)\n",
      "creating index...\n",
      "index created!\n"
     ]
    }
   ],
   "source": [
    "ground_truth_annotations = COCO(GROUND_TRUTH_ANNOTATION_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..then we will need to actually open the `predictions.json` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submission_file = json.loads(open(\"predictions.json\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and now we load the results from the predictions file using the `loadRes` function from the `cocoapi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading and preparing results...\n",
      "DONE (t=0.01s)\n",
      "creating index...\n",
      "index created!\n"
     ]
    }
   ],
   "source": [
    "results = ground_truth_annotations.loadRes(submission_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initiate Evaluation\n",
    "\n",
    "We initiate the evaluation by using the `COCOEval` class and instantiating an evaluation for segmentation between the `ground_truth_annotations` and the `results`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cocoEval = COCOeval(ground_truth_annotations, results, 'segm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compute and Accumulate Metrics\n",
    "\n",
    "This step might take a few minutes, so please be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=0.30s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.17s).\n"
     ]
    }
   ],
   "source": [
    "cocoEval.evaluate()\n",
    "cocoEval.accumulate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summarise Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000\n",
      "Average Precision : 0.0 || Average Recall : 0.0\n"
     ]
    }
   ],
   "source": [
    "average_precision = cocoEval._summarize(ap=1, iouThr=0.5, areaRng=\"all\", maxDets=100)\n",
    "average_recall = cocoEval._summarize(ap=0, iouThr=0.5, areaRng=\"all\", maxDets=100)\n",
    "print(\"Average Precision : {} || Average Recall : {}\".format(average_precision, average_recall))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
